Acknowledgements

This work has been co-funded by the European Union under the Horizon Europe vera.ai

grant 10039055, and the Swiss State Secretariat for Education, Research and Innovation

(SERI) under contract number 22.00245; TITAN project, Grant Agreement number

101070658; AI4Media project, Grant Agreement 951911; and AI4Trust project, Grant

Agreement 101070190.

Proofreading and formatting Joanna Wright, University of Sheffield.

Published February 2024

This white paper is made available in Open Access under a Creative Commons Attribution-

Non Commercial-No Derivatives 4.0 International Licence (CC BY-NC-ND)

Contents

Bibliography .......................................................................................................................... 29

Figures

Figure 1: Evolution of fully synthetic image quality over the years. Image originally tweeted by

Ian Goodfellow (creator of GANs), then extended by (Masood et al., 2023) and then extended

Figure 4: Screenshots of the aforementioned BBC, Telegraph and Tagesschau articles (1-3)

................................................................................................................................................ 7

Figure 5: Screenshots of the aforementioned Wired and BR24 articles (4-5) ........................ 7

Figure 6: Overview of MINTIME architecture that handles the detection of deepfakes in videos

with multiple identities. Image from (Coccomini et al., 2022). ............................................... 12

Figure 7: Depictions of Fourier transform (amplitude) of the artificial fingerprint estimated from

1000 image residuals from 10 recent synthetic image generation models, which illustrates the

the Kremlin chief propagandist Vladimir Soloviov on a fake video with the detection of text

blocks, the language identification and the possibility to send the block to a translation engine

in one click. ........................................................................................................................... 17

Figure 10: Screenshot of the interface of the synthetic media detector with a fake picture of

US president Joe Biden, allegedly sleeping during a G20 summit. ...................................... 18

As the fluency and affordability of generative AI continues to increase from one month to the

next, so does its wide-ranging misuse for the creation of affordable, highly convincing large-

scale disinformation campaigns. Highly damaging examples of AI-generated disinformation

abound, including highly lucrative Facebook ads

seeking to influence voters through

found to host ads with political deepfake videos, which used voice imitation (RTL Lëtzebuerg,

2023). Beyond videos, AI-generated images have been used to spread disinformation about

Gaza (France, 2023; Totth, 2023) and propagate divisive, anti-immigrant narratives (The

Journal, 2023). Audio deepfakes have also been reported by fact-checkers, so far, these have

The countries being targeted span the entire European Union (and beyond), including highly

susceptible countries such as Bulgaria (

Bossev, 2023; BNT, 2023), where citizens have low

levels of media literacy and critical thinking skills, as well as a lack of awareness of the

existence of sophisticated AI-generated images, videos, audio, and text.

Platform actions aimed at countering disinformation in posts and adverts have so far also fallen

short on detecting and removing harmful AI-generated content. All major social media

platforms and chat apps have been impacted. For brevity, here we include only some

examples from Facebook (ads (

2.2 AI-generated audio

Similar to text and visual synthesis, AI-based speech synthesis methods have made enormous

Here, text analysis, acoustic modelling and audio synthesis are no longer isolated, but

mapped, trained and optimised within a common network architecture. This network

architecture can also be trained with roughly annotated data (in the form of pairs of text and

audio recordings), which are available in almost unlimited quantities without the need for

complex expert annotation. Speech synthesis has been achieving ever better speech quality

over recent years, as early as 2017, often on par with natural speech, e.g. with approaches

transferring the unique characteristics of a speaker's voice from an audio recording onto

another set of spoken words. Notable examples in this area include

landscape of voice synthesis has been extended by the availability of pre-trained models of

publicly-known figures, exemplified by platforms like

While speech synthesis has emerged as a powerful tool for disinformation creation, its

potential for misuse (e.g. to damage citizen trust or undermine democratic processes) has

long been underestimated. A few examples include:

conversion technology to impersonate a high-profile individual in the context of the

falsely attributed to their broadcasts (

Figure 4: Screenshots of the aforementioned BBC, Telegraph and Tagesschau articles (1-3)

(

Meaker, 2023).

covered by BR24, emphasising the seriousness and potential harm of such deepfakes,

Dataset of Synthetic Speech (Yaroshchuk et al., 2023).

Papastergiopoulos et al., 2023). Particularly acute is the need for new research datasets that

of combinations of potential synthesis methods, the need for generalisability, and the

importance of explainability and interpretability.

Notable state-of-the-art speech synthesis detection approaches include an anti-spoofing

model for detection of speech attacks (

through emotion recognition (Conti et al., 2022); identification of specific artefacts in spoofed

speech (Tak et al, 2021); a spoofing attack detection system (Jung et al, 2022); and new

neural models for synthetic speech detection (Cuccovillo, Gerhardt & Aichcroft, 2023).

As with all other areas developing AI methods to detect synthetically generated content, there

is a strong potential for misuse, which in the case of research means striking a balance

Looking ahead, the primary challenge in the realm of synthetic speech detection is the

absence of a robust and sustainable ecosystem for research and development. Central to this

is the need for increased funding and incentivisation to create GDPR-compliant real speech

materials and datasets. These are essential for training and testing detection technologies and

must be significantly expanded to keep pace with the rapidly advancing commercial synthesis

sector, which benefits from greater funding. Without such expansion, European researchers

may find themselves primarily occupied with replicating existing speech synthesis methods

and generating data, rather than innovating in the field of detection.

Moreover, it is essential to have ongoing funding for research in two key areas. First, in

commercial applications are necessary. These models, complemented by public funding,

of synthetic speech samples.

3.3 Detecting machine-generated textual disinformation

Due to the potential for misuse of machine generated text (MGT) for influence operations

(

Goldstein et al., 2023), disinformation (Buchanan et al., 2021), spam or unethical authorship

(Crothers et al., 2022), there is a substantial amount of research on the detection of machine-

generated text (Jawahar et al., 2020; Stiff & Johansson, 2022; Uchendu et al., 2023a).

Researchers have shown machine-generated text is now so fluent that human readers cannot

distinguish it reliably from human-written text, with accuracy only slightly above random

et al., 2020). State-of-the-art methods for detection of machine-generated text include

Uchendu et al., 2020; Fröhling & Zubiaga, 2021; Kumarage et al., 2023). Typically

these are deep learning-based detectors (Zellers et al., 2020; Uchendu et al., 2021; Bakhtin

et al., 2019; Liyanage et al., 2022; Rosati, 2022), which however are susceptible to adversarial

perturbations (Gagiano et al., 2021; Crothers et al., 2022; Wolff & Wolff, 2022) and need a lot

of labelled data to perform well.

Statistics-based detection techniques are robust to adversarial perturbations and are

unsupervised, requiring minimal data (

Gehrmann et al., 2019; Mitchell et al., 2023; Gallé et

al., 2021; Su et al., 2023). However, while these statistics-based detectors are more robust,

they do not perform as well as deep learning-based models in some cases. Hybrid approaches

8

vera.ai projects has begun to address

this through the creation of the first comprehensive benchmark MULTITuDE of black-box,

statistical and fine-tuned machine-generated text detection methods in multilingual settings

using a novel MULTITuDE dataset, covering 11 languages and 8 SOTA LLMs (Macko et al.,

2023).

Our research experiments on this new dataset show that most currently available black-box

methods for detection of AI-generated texts do not work in multilingual settings and that the

statistical approaches lag behind the fine-tuned deep learning ones. The results (Macko et al.,

performing state-of-the-art detection methods suffer from a significantly degraded

performance on short text. The emergence of AI-driven bots on platforms such as X/Twitter

has made the development of more effective methods a particularly urgent challenge.

Professionals and Citizens

4.1 InVID-WeVerify plugin

Several AI-powered tools are integrated into the InVID-WeVerify plugin

9

, a browser extension

because it allows, in most of the cases, to debunk decontextualised or manipulated videos,

including the first generation of deepfakes (the ones made from already existing video

fragments). The keyframe service is based on a Convolutional Neural Network (CNN)

implemented by CERTH.

https://chromewebstore.google.com/detail/fake-news-debunker-by-

Figure 8: Screenshot of the keyframe fragmentation service (on a fake video during the Ukraine war

debunked among others by the British fact-checker FullFact).

media detector based on machine learning and forensic traces discovery that help end-users

to determine if an image has been created by AI.

Figure 10: Screenshot of the interface of the synthetic media detector with a fake picture of US

president Joe Biden, allegedly sleeping during a G20 summit.

and follow-up of the creation process to better understand the disinformation potential

threats thanks to interpreting and critically assessing the reasoning and arguments

To this end, the TITAN project

being put forward and the reliability of statements they encounter;

citizens to the risks.

materialise in the final text. The AI4Media project is monitoring and analysing the proposed

and upcoming legislation

2024.

publishers are opting out from generative AI bots scraping (

Tar, 2023). Some want to use this

as leverage for contracting licensing agreements. However, removing trustworthy news and

investigative journalism content will impact the quality of the data present in the dataset and

impact the general quality of the output. In other words, the removal of content from quality

news publishers from the training data of LLMs risks over-representation of the disinformation

content available online and exacerbation of the misinformation and hallucination tendencies

of generative AI models.

has been

sued for unauthorised use of copyright content for training its generative AI models (

Wolters Kluwer, October 2023).

28b). France has recently introduced law proposal n°1630 (

Assemblée Nationale No. 1630)

obtain an authorisation from the author (or IPR holder) before using copyright-protected

material for the development of an AI system. Further details can be found in this AI4Media

Competition law and power dynamic concerns

market concentration and competition issues. At the same time, content creators are trying to

protect themselves and the value of their work from AI imitations, through increased use of

tools like Nightshade and Glaze that prevent their work from being scraped (

which need further investigation.

5.2 Disinformation Risks Arising from Generative AI Usage

In common with other types of AI technology, generative AI suffers from a number of horizontal

issues: various kinds of bias and discrimination, media independence, inequalities in access

to AI, labour displacement, privacy, transparency, accountability and liability (

AI4Media, 2022;

AI4Media, 2023b). We next discuss some which are also specific to generative AI.

Manipulation and AI-anthropomorphism

of legal responsibility and liability; and a better balance between the precautionary principle

and the innovation principle. Education and awareness campaigns to better inform people of

the risks associated with AI systems are also needed. The risks of AI to fundamental human

rights should be first identified, analysed and mitigated, before the AI application is made

publicly available.